Method and apparatus for teaching prosodic features of speech

ABSTRACT

A system and apparatus for teaching prosodic features of speech senses and extracts prosodic or suprasegmental variables of a user&#39;s speech segment. Prosodic features of speech include pitch and loudness variations, as opposed to articulatory or sequential features of speech which are the primary determinants of phoneme variations. Once prosodic variables have been extracted from a speech segment, the variables are used to modulate a quasiperiodic waveform such as a sinusoid, a pulse-train, or a synthesized vowel-like waveform, or the parameters can be used to modulate a random-noise-like waveform. A modulated waveform can be played acoustically, and the user can hear the variation of the prosodic parameters without interference from the articulatory parameters of a complete waveform. This auditory feedback can be combined with visual feedback of the speech segment to teach proper prosodic speech formation. Auditory feedback for teaching prosodic features can also be formed without a modulation process by removing articulatory information from a speech segment, and non-acoustic measures sensors of prosodic feature, such as an electroglottograph, can also be employed.

This application is a divisional application of application Ser. No.08/448,708, filed May 24, 1995, now U.S. Pat. No. 6,109,923.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a system for teaching spoken languages. Moreprecisely, this invention relates to a system for teaching the prosodicfeatures of a spoken language.

2. Description of the Related Technology

The information in natural spoken languages is transmitted by acombination of articulatory variations and prosodic variations (orprosody). In English and most European and middle eastern languages, thearticulatory variations, often referred to as the segmental features ofspeech, generally determine which sounds of the language (phonemes) arebeing represented, and therefore also determine which words are beingrepresented. In such languages, prosodic variables, or suprasegmentalfeatures as they are sometimes referred to, such as sound intensity(loudness), intonation (the variation of voice pitch on the perceptuallevel, or voice fundamental frequency on a physical level), variationsin the rate of speech (usually referred to as the ‘duration’ variable),and voice source quality primarily determine the stress patterns in asentence and convey emotional factors and secondarily interact with thesegmental features to influence meaning. However, in the so-called tonelanguages such as Thai or Mandarin Chinese, the intonation pattern isalso an important determinant of the word being pronounced.

If the patterns of the prosodic features in a new language differmarkedly from those of a learner's previous languages, the propercontrol of prosodic parameters such as intonation and intensity may bedifficult to learn, especially for an adult learner. Thus the accent ofa foreign speaker often includes intonation and stress pattern errorsthat affect naturalness and sometimes understandability. In addition,there are many persons that require coaching in their native language toimprove their use of prosody during speech or singing.

To aid the learner of a new language, speakers with voice or speechproblems related to prosody, or deaf learners of an oral language, anumber of systems have been proposed to convert prosodic variables intoa feedback display that uses a sense modality other than hearing. Thevariables of intonation and intensity can be measured readily fromeither the acoustic speech waveform or, in the case of intonation, froma contact microphone on the neck, an airflow measuring mask, or a deviceknown as an electroglottograph. Examples of such devices are describedin Martin Rothenberg, Measurement of Airflow in Speech, 20 J. Speech andHearing Res. 155-76 (1977); Martin Rothenberg, A MultichannelElectroglottoaraph, 6 J. of Voice 36-43 (1992); and U.S. Pat. Nos.3,345,979, 4,862,503 and 4,909,261.

Although the tactile sense modality has been used for feedback,especially in speech training for profoundly deaf learners, such asMartin Rothenberg & Richard D. Molitor, Encoding Voice FundamentalFrequency into Vibrotactile Frequency, 66 J. of the Acoustical Soc'y ofAm. 1029-38 (1979), the visual sense modality is more commonly employed.Various systems for providing visual feedback for the teaching ofintonation are described by Dorothy M. Chun, Teaching Tone andIntonation With Microcomputers, CALICO J., Sept. 1989, at 21-46. On avisual display, the time variable is most often encoded into a spatialvariable, usually horizontal distance, and the intonation or theintensity of the voice signal is encoded into a visual variable such asvertical distance, line width, image brightness or image color value.The resulting visual display is usually displayed and held on the screenusing a memory function, for post-production viewing and analysis by thespeaker or a teacher and for possible comparison with a model trace ordisplay. The state-of-the-art may include scrolling of the display ifthe speech segment is too long for the display, and provisions for timeand amplitude normalization between the speaker's trace and the modeltrace.

However, the transformations between the auditory sensations of pitchand loudness (the primary perceptual correlates of intonation frequencyand acoustic intensity or amplitude, respectively) and visual variablesrepresenting intonation frequency and time, or amplitude and time, arenot easy or natural for many learners. The visual display oftenhighlights details that are irrelevant to auditory perception. Theseirrelevant details are sometimes referred to as microintonation. G. W.G. Spaai, et al., A Visual Display for the Teaching of Intonation toDeaf Persons: Some Preliminary Findings, 16 J. of MicrocomputerApplications 277-286 (1993). In addition, a major problem in the use ofsuch visual displays is the fact that natural speech contains many gapsin the voice stream caused by unvoiced sounds, certain occlusiveconsonants, and short pauses. Such microintonation and gaps in thestream of voice are readily ignored by the auditory system when judgingthe patterning of voice pitch or loudness in speech or singing. Therehave been a number of automated systems proposed for simplifying visualdisplays of intonation in order to circumvent these factors such as G.W. G. Spaai, et al., A Visual Display for the Teaching of Intonation toDeaf Persons: Some Preliminary Findings, 16 J. of MicrocomputerApplications 277-286 (1993); S. Hiller, et al., SPELL: An AutomatedSystem for Computer-Aided Pronunciation Teaching, 13 Speech Comm. 463-73(1993). However, a visual display constructed according to the presentart remains difficult to interpret without special training.

FIG. 1 illustrates this difficulty with a trace of an intonation patternthat might be typical for the English sentence “We put the apple in.”The time variable is measured on horizontal axis 50, and the intonationvariable is measured on vertical axis 60.

SUMMARY OF THE INVENTION

In this system, each prosodic variable that is the subject ofinstruction is superimposed or modulated onto either a periodic tonewaveform, or a noise-like signal, so that during a replay of thismodulated tone, or noise, the user can hear the variation of themodulating variable without interference from the articulatory featuresof the complete voice waveform. For encoding voice intonation, orcombined intonation and intensity, the effect might be similar tohearing a hummed or a single-vowel version of the original phrase orsentence. With this modulated tone or noise input, the user's auditorysystem can provide the user with the sensation of continuous or smoothpitch or loudness variation even in the presence of microintonation orgaps caused by unvoiced sounds, occlusive consonants, or short pauses.

Also, by replaying this modulated-sound version of the phrase orsentence simultaneously with a visual encoding of the prosodic variable,the user can learn to interpret the visual display correctly. Thislearning process can be helped by a slower-than-real-time playback insome cases, so that correspondence between the variation of the auditoryparameter and the visual display can be followed more easily.

It is also envisioned that the visually encoded patterns of the user canbe displayed in close proximity to one or more model patterns tofacilitate visual comparison. Model patterns can be recorded by aninstructor or pre-stored in a system memory, or generated from a systemof rules that describe the proper variation of that parameter in thelanguage being learned. A rating of the closeness of match of the user'spatterns to the model patterns can also be made available.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a trace of an intonation pattern.

FIG. 2 shows an embodiment of an apparatus for teaching prosodicfeatures of speech.

FIG. 3 shows an alternative embodiment of an apparatus for teachingprosodic features of speech.

FIG. 4 shows another alternative embodiment of an apparatus for teachingprosodic features of speech.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the embodiment shown in FIG. 2, one or more sensors 1, such as amicrophone, throat microphone, airflow mask, or electroglottograph,provides a signal 2 from which one or more prosodic variables, such asintonation frequency or sound intensity, can be extracted.

Prosodic variable extractor 3 extracts the prosodic parameters from thesignal 2 and provides an output 4 which is an encoding of the timevariation of the desired prosodic parameters. For the measurement ofintonation frequency, prosodic variable extractor 3 can be an electroniccircuit or an equivalent computer program for voice pitch-tracking thatis constructed according to current art, and which can consist of alow-pass filter that effectively reduces the amplitude of harmonicsabove the fundamental frequency, followed by a zero-crossing detectorand a mechanism for measuring the period of the time interval betweensuccessive positive-going or negative-going zero crossings, with thesuccessive, period-by-period estimates of intonation frequency beingoutput as the reciprocal of each successive period estimate. Thelogarithm of the reciprocal may alternatively be output in order tobetter equalize the variation range of the parameter display at low andhigh voice pitches. For the measurement of voice intensity, prosodicvariable extractor 3 can be an electronic circuit or equivalent computerprogram that computes and provides as an output 4 either the peak signalamplitude, average rectified signal amplitude, or RMS signal amplitude,over a moving time window that is preferably slightly larger than thelongest expected voice period. A logarithmic or other similar nonlinearscaling may also be used for the intensity parameter in order to improvethe appearance of the display.

The segment storage and replay device 5, which can be a segment of thedigital memory of a digital- computer and a computer program for readingthat segment of memory, receives the output 4 of the extractor 3, andstores the values of the variables over a finite period of time having aduration determined by an output 20 of a program manager 7. Programmanager 7 can be, in part, a computer program for controlling therecording and replay functions of the segment storage and replay device5, including the starting and stopping of the storage function, thestarting and stopping of the replay function, and the speed of thereplay. The program manager 7 can receive as one input the output 4 ofthe extractor 3 for use in determining the correct timing of the segmentstorage operation. The segment storage and replay device 5 canaccordingly be directed by an output 21 of the program manager 7 to playa stored segment of the variables, either once or repeatedly, to providean output 8. Additionally, the capability may be provided to store morethan one segment, in order that more than one segment can be outputsimultaneously for viewing on the video display 6, which can be thevideo monitor of a computer.

For optional real-time viewing of the display (while the user isspeaking), the video display 6 can receive the extractor output 4through a switch 25, which can operate under the control of the programmanager 7, with the switch 25 connecting the video display 6 to output 4of the extractor 3 during the extraction process and to output 8 of thesegment storage and replay device 5 during replay periods. The replayspeed of the variables at output 8 can preferably be adjusted by anoutput 22 of the program manager 7, including adjustment to a speedslower than real-time. Note that this adjustment of replay speed shouldpreferably be performed without a change in the auditory sensationproduced by the audio transducer 11, which may be an amplifier and aloudspeaker or an amplifier and headphones. If intonation is the prosodyparameter being taught, the effect of slowing the replay speed should bepreferably one of speaking slower and not of slowing the speed of a taperecorder playback.

Output 8 of the segment storage and replay device 5 is also sent to awaveform generator and modulator 9. Waveform generator and modulator 9can be a standard MIDI audio synthesizer, such as those commonlyprovided in computers conforming to the MPC standard for multimediacomputers, or it can be a specially-constructed computer program thatgenerates the required waveform. Optimally, the output of the waveformgenerator and modulator 9 is synchronized, through a timing signal 26from the program manager 7, with a marker on the trace of the videodisplay 6, such as an arrow, dot, line, or area of brightening or colorchange on the display, which traces for the user the instantaneousvalues of the visual display during a replay cycle. The waveformgenerator and modulator 9 generates a quasiperiodic waveform output 10,with possible waveforms including a sinusoid, a pulse-train, or asynthesized vowel-like waveform. The waveform 10 may also have arandom-noise-like nature if encoding an amplitude-derived variable fromoutput 4.

The form of the quasiperiodic waveform may be selectable from an output23 of the program manager 7. At least one parameter of the waveform 10is controlled or modulated in some proportion by the output 8 of thesegment storage and replay device 5. When encoding a voice intonationfrequency variable, the controlled or modulated parameter of outputwaveform 10 is preferably its frequency.

Otherwise, the output 10 of the waveform generator and modulator 9 issent to audio transducer 11. A waveform storage 12, which may be asection of the digital memory of a computer, may also be provided forstoring model waveforms representing correct pronunciation, the output13 of which may be input to the video display 6 and displayedsimultaneously with the user's waveform to facilitate a comparison ofthe pattern of the user's voice with the pattern of the model. Output 13may alternatively be connected as an input to the segment storage andreplay device 5 so that a simultaneous visual and auditory comparisoncan be made. This second configuration is shown in FIG. 2.

The program manager 7 can have a user input capability, which can bemanual switches or knobs, or can be areas on the display screen of thevideo display 6 that can be clicked on with the use of a computer mouse,so that one or more of the system parameters that are set by the programmanager 7 can be controlled through user input.

An alternative embodiment shown in FIG. 3 is constructed in a mannersimilar to the embodiment of FIG. 2, except that segment storage andreplay device 5 precedes the prosodic variable extractor 3, so that awaveform 14 of the original speech signal can be replayed, instead ofthe processed waveform 10, if so selected by switch 15. In thisembodiment, the sensors 1 preferably include a microphone. For optionalreal-time viewing of the display using this embodiment, areal-time/stored switch 30 may be provided that directs the sensoroutput 2 to the input of the extractor 3 during the recording process.

Another alternative embodiment shown in FIG. 4 is similar to theembodiment of FIG. 3 except that the acoustic waveform is obtained froma processed version of the original sensor waveform rather than by aprocess of modulation. This eliminates the need for the waveformgenerator and modulator 9 in FIGS. 2 and 3. In this embodiment, prosodicvariable extractor 3 includes an articulatory variation remover 40 thatremoves or attenuates any articulatory information or undesired prosodicinformation in the sensor output 2, while retaining the prosodicparameters to be taught. Prosodic variable measurer 41 follows theremover 40. Measurer 41 determines the values of the prosodic variablesat the output 31 of the articulatory variation remover 40 and providesthese measurements at an output 8, which is supplied to the videodisplay 6.

If a sensor 1 in the embodiment of FIG. 4 is a microphone, there are anumber of methods well-described in the literature for implementing thearticulatory variation remover 40, including the process known as LinearPredictive Coding (LPC analysis). In the case of LPC analysis, theoutput 31 could be the LPC residual signal, which contains littlearticulatory information. Another method is inversefiltering, in whichcase the output 31 during voiced speech would be an approximation of thelaryngeal voice source waveform. If the intonation range is not toogreat (less than about an octave), intonation frequency can also bepreserved by a band-pass filter encompassing the lowest and highestexeected intonation frequencies, in which case the output 31 would be anapproximate sinusoid at the voice intonation frequency. Each of thesethree methods produces a simple waveform at the voice intonationfrequency which can be easily standardized in waveform and intensitybefore being sent to output 31.

A method for removing spectral (frequency) information from a digitizedspeech signal and retaining only the amplitude of the waveform isdescribed by Manfred Schroeder, Reference Signal for Signal QualityStudies, 44 J. of the Acoustical Soc'y of Am. 1735-36 (1968). Using thismethod, each successive digitized sample of the waveform is given arandomly assigned polarity. The resulting waveform, which can be sent tooutput 31, would retain most of the intensity and timing information ofthe original waveform.

If a sensor 1 in the embodiment of FIG. 4 is an electroglottograph or athroat microphone (a vibration sensor attached to or pressing on thesurface of the neck), the sensor output 2 bears little or noarticulatory information, and the articulatory variation remover 40 forpreserving intonation frequency can be a bandpass filter encompassingthe lowest and highest expected intonation frequencies (about 50 to 300Hz for men and 75 to 450 Hz for women), followed by a system for forminga standard pulse for each cycle.

In the embodiment of FIG. 4, waveform 31 may be sent directly to audiotransducer 11 or optionally be passed through an acoustic waveformfilter 16 of a standard type that optimizes the acoustic quality of theresulting tone. The waveforms so formed may include pulsatile,sinusoidal, and vowel-like waveforms. As in the embodiment of FIG. 3, aswitch 15 may be included to allow the user to listen to the originalstored sensor waveform 14 instead of the processed waveform 33.

In each of the above embodiments, audio transducer 11 can also be avibrotactile or electrotactile transducer or an array of suchtransducers, so that this apparatus may be used by a deaf personlearning the prosodic features of a spoken language. For the learning ofvoice intonation through vibrotaction, the intonation frequency of thelearner and of a model voice is advantageously reduced to a rangeoptimal for the tactile sense, as described by Martin Rothenberg &Richard D. Molitor, Encoding Voice Fundamental Frequency intoVibrotactile Frequency, 66 J. of the Acoustical Soc'y of Am. 1029-38(1979).

This method and apparatus for teaching prosodic features of speech may,of course, be carried out in specific ways other than those set forthhere without departing from the spirit and essential characteristics ofthe invention. Therefore, The presented embodiments should be consideredin all respects as illustrative and not restrictive and allmodifications falling within the meaning and equivalency range of theappended claims are intended to be embraced therein.

I claim:
 1. An apparatus for teaching prosodic features of speechcomprising: a sensor having a signal sensor output; a prosodic variableextractor having a signal input connected to said signal senor output ofsaid sensor and having an output signal representing a prosodic featureof said signal sensor output; an audio transducer, connected to anoutput of the signal generator; and wherein the prosodic variableextractor includes an articulatory variation remover.
 2. An apparatusfor teaching prosodic features of speech comprising: a sensor having asignal sensor output; a segment storage and replay device having asignal input connected to said signal sensor output of said sensor andhaving a signal segment storage and replay output; a prosodic variableextractor having a signal input connected to said signal segment storageand replay output said segment storage and replay device and having anoutput signal representing a prosodic feature of said signal segmentstorage and replay device output; a waveform generator and modulator,connected to an output of the extractor; and an audio transducer,connected to an output of the waveform generator and modulator.
 3. Anapparatus for teaching prosodic features of speech according to claim 2,further comprising: a video display, connected to an output of theextractor.
 4. An apparatus for teaching prosodic features of speechaccording to claim, 2, wherein the extractor is also connected to anoutput of the sensor.
 5. An apparatus for teaching prosodic features ofspeech according to claim 2, wherein the audio transducer is alsoconnected to an output of the segment storage and replay device.
 6. Anapparatus for teaching prosodic features of speech comprising: a sensorhaving a signal sensor output; a segment storage and replay devicehaving a signal input connected to said signal sensor output of saidsensor and having a signal segment storage and replay output; a prosodicvariable extractor having a signal input connected to said signalsegment storage and replay output said segment storage and replay deviceand having an output signal representing a prosodic feature of saidsignal segment storage and replay device output; an acoustic waveformfilter, connected to an output of the extractor; and an audiotransducer, connected to an output of the acoustic waveform filter. 7.An apparatus for teaching prosodic features of speech according to claim6, wherein the extractor comprises an articulatory variation remover. 8.An apparatus for teaching prosodic features of speech according to claim6, wherein the audio transducer is also connected to an output of thesegment storage and replay device.
 9. An apparatus for teaching prosodicfeatures of speech according to claim 6, further comprising: a videodisplay, connected to an output of the extractor.
 10. An apparatus forteaching prosodic features of speech according to claim 6, wherein theextractor is also connected to an output of the sensor.